DOCUMENT RESUME 



IR 020 298 

Hsieh-Yee, Ingrid 

ERIC User Services: Evaluation in a Decentralized 
Environment . 

Educational Resources Information Center (ED) , Washington, 
DC. 

2000-04-21 

6 Op . ; This paper is part of a set of five papers 
commissioned by the U.S. Department of Education as 
preliminary preparation for an upcoming comprehensive 
system-wide evaluation of the Educational Resources 
Information Center (ERIC) . For the complete set of 
commissioned papers, see IR 020 296-299 and IR 020 101. 

ED - 9 9 - PO - 4 2 8 9 
Reports - Evaluative (142) 

MF01/PC03 Plus Postage. 

Access to Information; ^Educational Resources; ^Evaluation 
Methods; * Informat ion Services; Information Sources; Needs 
Assessment; Online Systems; *User Needs (Information) 

*ERIC ; *ERIC Clearinghouses; Question Answering 

This report presents findings from a study of the 
Educational Resources Information Center (ERIC) user services, describes 
changes to current user services that can help ERIC become more competitive, 
and discusses strategies and options for evaluating ERIC user services. Data 
show ERIC information professionals have done much to identify users' 
information needs, .made continuous promotion efforts to reach users and 
non-users, and provided an impressive array of services and products. They 
have adopted information technology to enhance their services and 
demonstrated strong skills in identifying, synthesizing, and organizing 
resources valuable to their constituencies. They also show unwavering 
dedication to serving users. But they are constrained by limited resources 
and concerned about how to ensure quality services in the face of dwindling 
support and rising user expectations. Many of the services provided by ERIC 
are important in the digital age, and ERIC information professionals have the 
expertise to excel in those areas. If the recommended changes are 
implemented, ERIC will become a more competitive player in the information 
arena. Data also reveal that few evaluation efforts have been made and very 
few of them have produced useful data for planning or improvement services. 
Nevertheless, meaningful evaluation can be conducted by extending earlier 
study instruments and statistics. The paper presents an evaluation framework, 
discusses the importance of validity, reliability, and generalizability in 
evaluation, and describes strategies to ensure successful evaluation in a 
decentralized environment like ERIC. Appendices contain questions for ERIC 
Directors and questions for ERIC User Services Coordinators. (Contains 56 
references.) (Author/AEF) 



ED 437 935 

AUTHOR 

TITLE 

SPONS AGENCY 

PUB DATE 
NOTE 



CONTRACT 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 

ABSTRACT 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



ROZOZRR 



ERIC User Services: 

Evaluation in a Decentralized Environment* 



m 

ON 

r- 

m 



s 



by 



Ingrid Hsieh-Yee 
Associate Professor 

School of Library & Information Science 
Catholic University of America 
Washington, D.C. 20064 
E-mail: hsiehyee@cua.edu 
Phone: (202) 319-6270 
Fax: (202) 319-5574 



April 21, 2000 



U.S. DEPARTMENT OF EDUCATION 

national institute of education 

educational resources information 
CENTER (ERIC) 



m This document has been reproduced as 
received from the person or organization 



originating it. 

□ Minor changes have been made to improve 

reproducti on quality. ____ 

• points of view or opinions stated in this docu- 
ment do not necessarily represent official NIE 



position or policy. 



* This project has been funded at least in part with Federal funds from the U.S. 
Department of Education under contract number ED-99-PO-4289. The content 
of this publication does not necessarily reflect the views or policies of the U.S. 
Department of Education nor does mention of trade names, commercial 
products, or organizations imply endorsement by the U.S. Government. 




2 



BEST COPY AVAILABLE 



ERIC User Services: Evaluation in a Decentralized Environment 
TABLE OF CONTENTS 

Executive Summary 111 

I. The environment 

II. Purposes 

III. Assumptions 4 

IV. Methodology 5 

V. Findings ° 

Survey responses ° 

A. Users ° 

B. Question/Answering services 1 1 

C. Search services 

D. Web: Potential & challenges 17 

ERIC components’ evaluation methods 20 

VI. Discussion 22 

User services 22 

Evaluation efforts 28 

VII. Evaluation of ERIC user services 30 

Benefits of evaluation 30 

Evaluation cycle 3 1 

Factors affecting user services 32 

Strategies for evaluating in a decentralized environment 33 
Evaluation literature 34 

Evaluation types and levels 36 

Evaluation of ERIC user services: Three approaches 37 

Output analysis 37 

Usage analysis 38 

Impact analysis 40 

VIII. Conclusion 42 

44 

References 

Figure 1 . Evaluation Cycle 30 

Figure 2. PPE Method 31 

Figure 3. ERIC User Services Evaluation Options 52 

Appendix 1 . Questions for ERIC Directors 53 

Appendix 2. Questions for ERIC User Services Coordinators 55 



Ill 



Executive Summary 

This report presents findings from a study of ERIC user services, describes changes to 
current user services that can help ERIC become more competitive and discusses strategies and 
options for evaluating ERIC user services. Twenty questions in the areas of "users," "question and 
answering services," "search services," and "web potential and challenges" were used to solicit 
input from directors of sixteen ERIC clearinghouses and three support components. Five questions 
were also sent to all ERIC user services coordinators for their views on current services. 

Data show ERIC information professionals have done much to identify users' information 
needs, made continuous promotion efforts to reach users and non-users, and provided an 
impressive array of services and products. They have adopted information technology to enhance 
their services and demonstrated strong skills in identifying, synthesizing, and organizing resources 
valuable to their constituencies. They also show unwavering dedication to serving users. But they 
are constrained by limited resources and concerned about how to ensure quality services in the face 
of dwindling support and rising user expectations. Many of the services provided by ERIC are 
important in the digital age and ERIC information professionals have the expertise to excel in those 
areas. If the recommended changes are implemented, ERIC will become a more competitive player 
in the information arena. 

Data also reveal few evaluation efforts have been made and very few of them produced 
useful data for planning or improvement of services. Nevertheless, meaningful evaluation can be 
conducted by extending earlier study instruments and statistics. The paper presents an evaluation 
framework, discusses the importance of validity, reliability, and generalizability in evaluation and 
describes strategies to ensure successful evaluation in a decentralized environment like ERIC. 




IV 



Three approaches to user services evaluation-output analysis, usage analysis, and impact analysis- 
-are described and examples of how they may be conducted in the ERIC context are provided. 
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ERIC User Services: Evaluation in a Decentralized Environment 



I. The Environment 

One of the organizing principles of ERIC is a decentralized structure that consists of 
multiple clearinghouses, each focusing on a major area of the field of education; adjunct 
clearinghouses; affiliate clearinghouses; and support components. This model has served ERIC 
well over the years in expanding its coverage, reaching many constituencies, and delivering 
resources to users. As the 1999 ERIC Annual Report indicates, the ERIC system has set ambitious 
goals for the year 2000. It aims to provide easy and affordable access to ERIC resources in all 
formats to various audience groups in the education community through a variety of information 
dissemination channels. It seeks to expand its coverage and reach out to new users. It also intends 
to incorporate more Web technology to enhance its services. Data in the Annual Report reflect the 
commitment of the clearinghouses and the component services to work toward these objectives. 
The decentralized model has given ERIC components freedom to experiment with new services 
and new technology, and the array of products and Web sites developed by all ERIC components 
demonstrate the creators' ingenuity and dedication to users. 

The decentralized model, however, also has caused the system to appear fractured. For 
instance, the clearinghouses' web sites are rich with valuable resources, but links to the ERIC 
system (http://www.accesseric.org) and AskERIC are not consistently provided. Nor do the web 
sites provide links to other clearinghouses that may have information relevant to their users' needs. 
At some sites a visitor would need to be persistent to find information on a clearinghouse's 
relationship with the central ERIC system. The visitor also may need to know of the existence of 
another clearinghouse to obtain relevant resources from that clearinghouse. Furthermore, the 
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number of interfaces to the ERIC database, the large number of ERIC components, and the 
different look and feel of components' web sites all contribute to users' confusion over the ERIC 
system. More coordination is needed to give users a more coherent picture of the system. This is 
one of the easier tasks to enhance ERIC user services. Giving users a clear map of what the ERIC 
system offers and how a particular clearinghouse relates to the central system and other 
components will enable users to navigate the system and take full advantage of ERIC services and 
products. 

On the surface the decentralized structure may pose challenges for systematic evaluation of 
user services across ERIC components. But data on ERIC user services reveal more similarities 
than differences in the services provided by the components and suggest that evaluation of user 
services in a decentralized environment can be done systematically to give individual 
clearinghouses data for management and strategic planning. If comparative analysis of 
clearinghouse performance is desired, that will be possible as well. 



II. Purposes 

The objective of Paper #4 is to collect and analyze information on ERIC user services in 
anticipation for future evaluation of ERIC. This purpose entails taking an inventory of current user 
services and examining how these services have been evaluated, and with what success. In 
addition, ERIC Central is interested in information on what changes may be needed to make ERIC 
more competitive in the digital age, such as adding, dropping, or modifying some user services. 

Traditionally "user services" refers to reference services that are provided in-person, by 
telephone, or by mail. But "user services" has a much broader scope in the ERIC system. As 
reported by ERIC directors and user services coordinators, user services encompass 
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• Reference services (or Q&A services) provided in person, by phone, by e-mail, or on the 
Web. 

• Database and Web searches. 

• Referrals to other information sources or organizations. 

• Current awareness services. 

• Training workshops. 

• Conference presentations and exhibits. 

• Access to documents in a variety of formats. 

• Preparation of digests or bibliographies on special topics. 

• Preparation of search aids, tutorials, etc. 

• Compilation of electronic pathfinders. 

• FAQs compilation and management (fairly new service). 

• Management of listservs (fairly new service). 

• Design and maintenance of Web sites. 

Some of these services such as digests, newsletters, bibliographies, and pathfinders can be 
considered information products and may be analyzed in Paper #5. But since they are important 
services created in response to user needs, they are included in the analysis here. 
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III. Assumptions 

Evaluation is an important management tool that can help us identify current strengths and 
limitations in services, suggest ways to improve them, and help us plan for future services. 
Environmental forces and financial pressures all play a role in how an evaluation is conducted. In 
preparing the paper the investigator made a number of assumptions about the ERIC environment: 

• The ERIC system will continue to operate according to the decentralized model. 

AskERIC has demonstrated how a centralized question/answering service can enhance 
ERIC QA services, and new Web technology has enabled ERIC components to operate 
more efficiently, but the decentralized model is likely to continue because subject 
specialists at ERIC components have contributed to AskERIC’s success, and each ERIC 
component, with its special subject focus, is best positioned to meet the needs of its target 
audiences. 

• The principles of a new operations framework for ERIC, as presented in Rising 
expectations, will be followed. They are 

— "ERIC's capacity to meet rising expectations must be sustained and increased." 

— "ERIC must serve as a major load-bearing wall as the NLE is built." 

- "ERIC must become 'a national union catalog' of educational resources -in other words, 
the main venue for organizing, linking, describing, and making accessible all key education 
resources in all formats." 

- "ERIC must further develop its historic capacity to create useful and authoritative 
syntheses, analyses, and interpretations of education information." 
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• The ERIC system will actively adopt Web technology to enhance services. 

This assumption means that ERIC information professionals will continue to use new 
information technologies to enrich their services and even expand their information services 
on the Web. 

• The ERIC system is interested in expanding its user base and meeting the information 
needs of all types of users. 

The assumption means ERIC information professionals will expand their promotion efforts 
and their effort to monitor information needs of users. 

• The ERIC system plans to use evaluation data to help it manage the system, justify support 
for the system, and develop plans for future ERIC services. 

• The ERIC system intends for the evaluation data to assist clearinghouses and support 
components in planning and improving their services. 



IV. Methodology 

A communication from Keith Stubbs, former Director of ERIC, presents a list of questions 
and special issues for the commissioned paper writers. To address the topics for Paper #4, the 
investigator solicited input and assistance by emailing questions to directors ERIC clearinghouses 
and support components. The investigator also reviewed annual reports from all ERIC components 
to understand their operation. In addition, earlier evaluation reports (Design, 1978, ERIC user 
survey, 1991 ; Fry, 1972; Heinmiller, 1981; Stalford, 1990) and ERIC user services manual 
developed by Wagner (1997) were reviewed to understand previous evaluation efforts. All ERIC 
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Web sites were examined for their contents and structure. The literature on public services was also 
surveyed to identify evaluation methods and models. 

Twenty questions were posed to ERIC directors, covering four broad areas: Users, 
questions/answering services, search services, and the potential and challenges of the Web (see 
Appendix 1). In addition, the investigator sought direct input from user services coordinators on 
user services issues (see Appendix 2) to offer them an opportunity to add their own insights. The 
questions were sent to the directors on October 18, 1999, and data collection concluded on Dec. 7. 
Of the sixteen clearinghouses and three components, two responded to both sets of questions, 
sixteen responded to the questions posed to the directors, and one responded to the questions posed 
to user services coordinator. The last three months of 1999 was a very busy time for ERIC 
components, but all of them took time to provide information on their services and share their 
views on many issues with the investigator. Their generous support is a clear indication of their 
strong interest in evaluating user services. 



V. Findings 

This section summarizes data on user services and presents findings by the survey 
questions, which are presented in italic below. A description of service evaluation effort follows. In 
the next section the implications of the findings are discussed. 

• SURVEY RESPONSES 

A. USERS 
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Al. USERS: Do you have information on your users in terms of their types, purposes of using your 
services, frequency of usage, access methods (e-mail, phone, and/or Web), satisfaction with your 
services, and measurable impact of your services? 

USER TYPES AND NEEDS: Most ERIC components reported that they have kept statistics on 
user types and access methods as part of their contract agreement. These types of statistics are 
regularly included in the annual reports. Many have conducted surveys to gather information on 
users' reason(s) for using the system and their satisfaction with the services. Several ERIC 
components have analyzed user requests (paper and electronic) to identify trends and new topics of 
interest. Some have responded to identified needs by creating new products such as 
minibibliographies, electronic pathfinders, or digests. Some components have monitored listserv 
discussions, news reports, or relied on advisory panel to identify new research interests for them. 

PURPOSE: Some components said they have inferred the purposes from AskERIC requests or by 
categorizing service requests. But most ERIC components have not systematically collected users 
purposes for using their services. 

FREQUENCY: This information has not been collected consistently. Some indicated they have 
recognized repeat users who call regularly or by e-mail address, but no tracking has been done. 
Several Web surveys now ask users if they are repeat users. Service components such as EDRS has 
kept information on standing order customers and electronic subscription customers (starting in 
] 999) and on on-demand customers as to how they make contact, how many customers there are, 
and how many documents have been ordered by media type. 

SATISFACTION: This information was not collected regularly either. Respondents said user 
satisfaction usually have come in the form of thank-you notes or emails; some respondents have 
kept records of such messages while others have not. Recent Web surveys by clearinghouses asked 
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about overall satisfaction with ERIC services and products, and most respondents reported 
satisfaction. A recent AskERIC survey also found most users satisfied with the services (1998). 

This finding was comforting and important to some directors because a large percentage of their 
customized services came from AskERIC and they did not collect satisfaction measures for their 
services. Some ERIC components have included feedback cards in their service packages, but the 
response rate has been low. Many respondents are keenly aware of the importance of this measure 
and some have planned to include questions on user satisfaction in their future surveys. 

IMPACT: Some respondents reported that anecdotal information in letters and thank-you notes 
suggested that they had done a good job. Some directors attributed an increase in user requests to 
good services. But no ERIC components have been able to track service impact systematically 
because of limited resources. 

Many respondents indicated they are fully aware of the importance of evaluation but the 
lack of federal support has limited their abilities to measure the success or the values of their 
services. 

A2. NEW USER TYPES: Have you tried to reach out to non-traditional users such as users with 
disability and parents of home-schooled children? If so, how have you reached out to them? What 
lessons can you draw from such experience? 

Most respondents have used Web sites, workshops, and materials prepared for the target groups to 
reach non-traditional users. Several directors pointed out that their users ARE non-traditional users 
such as parents, immigrants, the disabled, the gifted, and urban and minority families. Some have 
used listserv announcements and mailings to partners and local liaisons to reach non-traditional 
users. Some have provided a text version of their site to make access easier, while several ERIC 
components have made their sites ADA compliant. A number of clearinghouses also have installed 
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TTY or TDD lines to support the disabled. Many commented that working with non-traditional 
users have led to wonderful results. The National Parent Information network, which forms 
partnership with several local groups in Illinois has demonstrated that quality information services 
can be provided by combining local resources and the resources of a national information service 
like ERIC. 

A3. NON-USERS: Do you know of users who could benefit from ERIC services but have not used 
the ERIC system? If so, how do you learn of them? Do you know why these users have not tried 
ERIC? What do you believe are the best ways to reach non-users? 

Many respondents have made serious efforts to promote ERIC services and products. In addition to 
more traditional approaches such as conference exhibits, presentations, and distributing ERIC 
publications through partners and state liaison networks, they have enriched their web sites with 
quality resources and more full text products to attract users. Several respondents have used 
listservs to announce ERIC products and services. Conversations at conferences and personal 
networks have also been used to spread the words about ERIC. 

Despite such efforts, there are still non-users out there. Respondents speculated that the reasons 
users do not try ERIC might include 

1) users have never heard of ERIC, surprising as that may seem; 

2) users do not perceive ERIC as relevant to their work; 

3) users know ERIC as a database and are not aware of services provided by the 
clearinghouses; 

4) users do not have access to ERIC CD-ROM or to the Internet version, 
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5) users are overwhelmed by the variety of interfaces to the ERIC database and the large 
number of clearinghouses; 

6) users are interested in instant or near-instant access to material and ERIC has not been 
able to offer them that option, especially in the area of full text. 

Some pointed out AskERIC have introduced ERIC to many new audiences. They also credited the 
Internet with making it easier to publicize their services. Several respondents recommended that 
ERIC make a concerted and continuous effort to have the entire ERIC system indexed on the 
Internet and make searching and retrieving information, especially full texts, as easy as possible. 
Several expressed concerns that the large number of search interfaces and engines have confused 
users. 

A4. USER NEEDS: How do you learn about the needs of your users? 

An effective way to understand user needs is to examine what users ask for, and all respondents 
have conducted such analysis on service requests. Some have analyzed questions quarterly, while 
others have done that in a less formal way. Some respondents have used the analysis results to 
design new publications, workshops, digests, bibliographies and other products to benefit a larger 
audience. Respondents also have relied on conversations at conferences, conference programs, 
popular literature, partners and state liaison network members, advisory committee, news report, 
listservs, and newsgroups to identify emerging trends and new topics. 

A5. UNMET NEEDS: Based on your experience , what needs of users remain to be met by your 
center? 

Respondents seemed to have a good idea of what users need. Full text products, practice-oriented, 
pre-packaged materials for practitioners, and special information packages for their target user 
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reminded us that user needs should be monitored continuously. 



1 1 



B. OUESTION/ANSWERING SERVICES 

Bl. QA Services: How have you provided QA services? Please include all the methods you have 
used. If you have evaluated these methods , please share your evaluation results with me. 

Although each clearinghouse has its own subject area and user groups, the services they provide 
are very similar. Question/answering services continue to be a major part of user services, and 
many have provided QA services both directly and through AskERIC. In some cases requests 
through AskERIC constituted 60 to 70 percent of clearinghouses' QA services, while others 
received more requests directly. All clearinghouses provided QA services by phone, by mail, by e- 
mail, through the web, and in person. Evaluation methods and results are discussed in the next 
section. 

B2. Comparison of QA services methods: Do you agree that QA services by e-mail , by phone, in 
person, or by the Web work equally well? If not, what types of questions can be best addressed by 
e-mail? What types of questions are most suitable for Virtual Reference Desk? 

Many respondents indicated that what works best depends on the question and the requester. Some 
said they prefer phone requests and in-person requests because these methods allow for reference 
interviews. But respondents also acknowledged that many users prefer e-mail and the Web for their 
convenience. Many respondents characterized questions suitable for e-mail requests as those that 
are clearly stated and posed by a user with some knowledge of the scope of the clearinghouse or its 
products. And questions appropriate for virtual reference desk are likely to be routine questions 




16 



12 



that do not require much subject expertise—FAQs were cited as an example. Some respondents 
expressed concern that AskERIC guidelines discourage them from contacting the requester to 
understand the requests when some of the questions clearly need clarification. 

It should be noted that respondents were not clear about the concept of virtual reference desk. 

Some interpreted it as reference services online or on the Web, and thought pre-prepared answers 
such as FAQs could be used to handle some typical questions. But the Virtual Reference Desk 
(VRD) special project in ERIC links users to digital inquiry services that provide subject experts to 
answer specific subject questions. 

B3. AskERIC vs. individual clearinghouse services: Have you provided your clearinghouse user 
services by e-mail? In your assessment, how helpful is AskERIC for your users? Can AskERIC 
replace individual clearinghouse user services or referral to services and resources in local 
libraries? 

This question is of special interest to Central ERIC and was posed to understand how AskERIC has 
contributed to ERIC services. All clearinghouses have answered questions referred by AskERIC. 
Some believed AskERIC have done a great awareness service for ERIC's services and products. It 
also has complemented individual clearinghouse user services and brought more users to the 
clearinghouses. But some were unsure about its value. No one felt AskERIC should replace 
individual clearinghouse user services for a number of reasons. First, AskERIC draws on the 
expertise within individual clearinghouses to answer complex service requests. Second, not all 
service requests are received through the Web. Third, user services encompass more than QA 
services. Finally, many service requests do not need a search and web site referral as recommended 
by the AskERIC guidelines. ERIC directors discussed the appropriate relationship between 
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individual clearinghouses and AskERIC in the 2000 National ERIC Joint Directors/Technical 
Meeting in March 2000. This issue is examined in VI. Discussion. 

B4. How do you feel about providing user services through Virtual Reference Desk? Do you plan 
to establish VRD in the near future? Do you have any concerns? If so, what are they? 

The concept of “virtual reference desk” remains fuzzy for many respondents. Some of them 
reported no plan to implement a VRD because of limited resources, while others interpreted VRD 
as. Web services and felt their highly organized sites that are rich with resources and experts’ views 
(presented in digest or other format) could qualify as VRDs. But as indicated above, VRD in the 
ERIC context does not answer questions. Instead, it connects users to experts who work at digital 
inquiry services. It may be a good idea for clearinghouses to add links to digital inquiry services to 
give users the option of reaching experts in the field. Such links, in essence, will enrich the ERIC 
system by adding practitioners and field experts as resources. But several issues need to be 
addressed before this idea can be implemented. First, what is the best way to link practitioners and 
experts to users? Is linking digital inquiry services the most efficient way for this purpose? If so, 
should the links be centralized in the current VRD or distributed among the components? The 
location and management of the links will determine where funding for such a service will be 
provided. And if links to digital inquiry services are not considered the best way to include 
practitioners and experts into ERIC, the ERIC community will need to explore other options. 

B5. Who among your staff provides QA services? How are they prepared? Do you have standards 
or guidelines for providing QA services? How do you ensure the quality of the services provided at 
your center? 

Most user services have been provided by user services coordinators or reference librarians, with 
assistance from graduate assistants. Many coordinators and reference librarians have MLS degrees. 
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some with subject background, while others have relied on their director for subject expertise. 
Coordinators and librarians believed that their professionalism and high performance standards 
have helped them ensure the quality of their services, and many of them do not have in-house 
guidelines or procedure for user services. Evaluation of user services staff have not been done 
regularly in some clearinghouses. 

Since AskERIC has employed part-time students and has the need to interact with a large 
number of user services coordinators and librarians, they have developed protocol and procedures 
for training their staff and ensuring consistency in service. ERIC/IT also has trained staff of other 
clearinghouses through the Internet on how to use AskERIC. Several respondents reported they 
have been more flexible in answering requests directly submitted to them, but in completing 
requests referred by AskERIC they have followed AskERIC guidelines and the responses tend to 
be more structured. System Guide to AskERIC Question Answering was developed in 1992, and has 
been revised in consultation with user services personnel from other clearinghouses; the electronic 
version is updated continuously. AskERIC also manages the Q-A net, a listserv, for information 
specialists to share strategies and communicate as necessary. 

To ensure the quality of answers to questions referred by AskERIC, all responses to such 
service requests have been copied to ERIC/IT, the administrator of AskERIC. AskERIC s question 
receiving and distribution system will be enhanced by a customized system, which will enable an 
AskERIC administrator to "maintain and enhance electronic systems and mechanisms for storing 
customer questions, tracking referrals and AskERIC responses, analyzing information including 
trends in topics, distribution of referrals, and response time by each clearinghouse in a more 
efficient manner" (from the AskERIC response to the survey). 
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B6. Have you evaluated your QA services? If so, please share your evaluation results with me. In 
addition , please indicate how satisfied you are with the evaluation methods. Are there things you 
would like to change in your next evaluation? 

Survey by mail, e-mail or the Web is the most common way to evaluate QA services. Some 
clearinghouses started to conduct Web surveys recently, while others had experimented with 
several methods with varying success. Most of the survey responses received were positive, 
showing users were reasonably satisfied with user services. User comments in a study by the 
National Parent Information Network reflect strong satisfaction with the services, and the two 
AskERIC surveys also reveal high satisfaction among users. But several respondents reported very 
low response rates for their surveys (in one case lower than 3%), and cautioned that their findings 
should not be generalized. One respondent commented that users were willing to check off boxes 
but reluctant to write a paragraph and satisfied users also offered few concrete suggestions to 
improve services. Another respondent cautioned that evaluation should be brief and kept to a 
"window period" so that users would not be so annoyed by studies that they ended up avoiding 
using ERIC services. A general trend was to treat e-mails and thank-you notes as indicators of 
good service and not to perform formal evaluation regularly. This practice may change in the future 
if the entire ERIC system begins to address evaluation seriously. 

Many respondents said they gained some understanding of users and their purposes of using 
ERIC from the two AskERIC surveys. Some also relied on the satisfaction rates reported in the 
latest surveys to assess the overall quality of their user services. Some clearinghouses have 
mounted feedback form on their web sites, but there is not enough data to gauge the effectiveness 
of this approach. One respondent reported a small number of responses have been received. One 
clearinghouse conducted a listserv survey and planned to analyze the data later. This clearinghouse 
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experimented with several evaluation methods and reported that in future surveys they would 
follow up on nonrespondents to determine if they are different from respondents. 

Increasingly, ERIC components are going beyond QA evaluation and relying on the Web to 
gather data on users unobtrusively. One clearinghouse planned to "explore the use of user profiles, 
customizable Web pages, and standing queries" to serve users and to gather data on their needs and 
behavior unobtrusively. Another has included counters on their web site and plans to use web logs 
and counter data to determine which parts of the site are accessed and how often. 



C. SEARCH SERVICES 

Cl. User training: If you train users to search ERIC, how do you do that? Do you use a tutorial? Is 
it a one-on-one instruction session? Do you assess how well the users learn to search ERIC? 

Training workshops tend to be offered at conventions and conferences. Many clearinghouses have 
offered individualized training workshops and one-on-one instruction as well. Most training 
programs are lectures, but some have included hands-on practice, and some have used online 
tutorial. Web tutorials developed by ERIC/Higher Education and ACCESS ERIC have received 
praises from respondents. Very few clearinghouses have assessed their training programs. 

C2. Do you log users' online searches and analyze them for search patterns and hot topics? Do you 
keep a log of activities at your Web site and analyze them for user behavior and needs? If so, 
please share your findings with me. 

Search log analysis was performed by very few respondents. Many showed interest in Web log 
analysis and hoped software would enable them to learn more about user behavior and use of the 
system. Some clearinghouses have search engines for their sites but have not analyzed their logs. 
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Instead, they have reviewed service requests for new topics and user needs and collected usage 
statistics of their web sites. 

C3. How do you feel about the current search engine and search interface for ERIC? Are there 
changes you M'ould like to make? 

Several respondents pointed out the proliferation of search engines and the number of interfaces of 
ERIC (online versions, CD-ROM versions, interfaces by commercial vendors, ERIC-sponsored 
interfaces) could be overwhelming for users. In addition to asking ERIC to standardize the 
interface, respondents would like to add the following features to the ERIC database: Online 
thesaurus, more seamless interface with the thesaurus and the search engine, more display options, 
better downloading and printing capabilities, and full access to thesaurus for all software versions 
of ERIC. 

C4. How do you feel about the current ways ERIC delivers search results to users? Are there 
changes you would like to make? 

This question was designed to solicit comments on the search output display of the ERIC database 
and the reliance on microfiche to deliver materials to users. The question turned out to be 
ambiguous and generated a wide range of answers. A few ideas worth noting include the 
suggestion that output should be sorted by a user-specified criterion and the wish that search 
citations should be accompanied by hotlinks so that the user can retrieve full text easily. 



n. WEB: POTENTIAL & CHALLENGES 

DI. Web technology offers a lot of possibilities for new types of user services. Please describe up 
to three new types of user services you have provided through the Web. In addition, please 
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describe two new user services you would love to provide through the Web if you have the 
resources. 

All ERIC components have been active in exploiting Web technology to enhance their services, 
and the services and products reflect their efforts! Direct e-mail reference services, links and 
referrals to related organizations, links to related resources, online tutorials, pathfinders, pathways, 
digests, full text products, listservs, online newsletter, FAQs, and so on. Several expressed interest 
in setting up listservs for their users, compiling FAQs, adding a search engine to their site, or 
adding audio-video clips to accompanying training modules. Overall, respondents reported a 
willingness to do more syntheses, add more full text of products produced by others such as lesson 
plans and develop web-based training to assist users in searching ERIC 

D2. Internet has brought users around the globe to ERIC. What are the challenges in serving 
international users? How have you addressed some of these challenges? What are the areas you 
would like the ERIC system to address? 

International users have always been a major user group for the ERIC system. As the Web 
becomes the primary medium for information access and dissemination, access by international 
users has increased. In fact, one clearinghouse found foreign domains constituted 13.5 percent of 
their Web access. One of the main concerns of respondents was about the appropriateness and the 
cost of such services. Some respondents felt ERIC should focused less on the United States. Some 
felt they have relied on international sources for documents, so it seems appropriate to reciprocate 
the services either by extending services to them or forming partnership with similar services in 
foreign countries. But others worried international users may be taking up too much of limited 

resources. 
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In addition, respondents cited language barriers, high user expectations, unreasonable 
expectations (such as receiving full texts at no cost) as challenges. If ERIC includes international 
audience as its users, the financial issues will need to be addressed first, then policy and procedures 
can be established to ensure these users' needs are met. 

D3. Some have expressed a wish to offer ERIC services anytime anywhere. Do you share that 
vision? What will it take to realize this vision? 

The concept of offering ERIC services anytime anywhere was interpreted differently by 
respondents. Some considered it a promise of synchronous services at all times and worried about 
the financial implications of such services, whereas others viewed it as a vision of asynchronous 
services and believed that Web sites have made ERIC components' services available anytime 
anywhere. 

D4. Some believe that ERIC staff should go beyond just providing answers and information to 
users. They feel that teaching users to search for information and evaluate information will serve 
users better in the long run. How do you feel about this view? Will the Web make it easier for us to 
teach users? 

All respondents agreed that teaching users is important and almost all respondents said the Web 
has made it easier for us to teach users. 

D5. Web technology raises user expectations of ERIC. Please describe what you have observed 
about user expectations. 

Several respondents observed that users seem to expect ERIC information professionals to do all 
their research for them. They expect to receive resources in all formats, whether they are publicly 
or privately available. They want immediate response. They expect answers to any questions at no 
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cost. They expect results in a user-friendly format and want full text products. In response, many 
clearinghouses have made materials under their control (such as their digests) available online. 
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• ERIC COMPONENTS’ EVALUATION METHODS 

As mentioned above, ERIC components have conducted formal and informal evaluation of 
services with mixed success. The evaluation methods used to analyze ERIC services and products 
include survey by mail, Web survey, phone survey, informal interview or focus group discussion, 
response card to publications, feedback card to answer packages, conference/workshop evaluation 
form, Web feedback form, Web log, and search log. 

Evaluation of products and services tend to focus on users' response, so it is understandable 
that survey is the most common approach. Almost all ERIC components have conducted formal or 
informal surveys by postal mail, and more and more ERIC clearinghouses have mounted surveys 
on their sites. Several Web surveys have the same questions, which ideally could generate data for 
comparative analysis, but most of the questions do not generate useful data for assessing a 
particular service or product. For instance, a question posed to users who have used ERIC services 
or products before says: "How satisfied have you been with the help or products you receive from 
us?" The answer could indicate a patron's general impression of ERIC but not enough to help us 
identify a good service or pinpoint problem areas. This is the characteristic of macroevaluation, 
which measures how well a system operates but is not able to offer explanation for why it operates 
at that particular level of effectiveness. Depending on the purpose of the evaluation, such data 
could be helpful, but they provide no basis for system improvement or future service plan. 

One ERIC component attempted a telephone survey to assess user satisfaction and found 
the method to be too costly and ineffective. Other components checked with users who place their 
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requests by phone about their satisfaction with the service, but that was done informally and 
infrequently. To obtain user response ERIC components added response cards or feedback cards to 
publications, answer packages, or document shipments to users. But the response rate tended to be 
very low. Some respondents interviewed users over the phone or at conferences, but these were 
done informally. 

As part of their effort to reach audiences, staff at clearinghouses made many presentations 
at conferences or offered workshops to users. Some talked to attendees to get feedback, while 
others included workshop/presentation evaluation forms. But the response rate to workshop 
evaluation was usually low. This may have something to do with the design of the evaluation form. 
For instance, one evaluation form contains three open-ended questions about a workshop. Such a 
design may not be appropriate for the setting because conference attendees tend to be in a hurry to 
get to another session or meeting and generally will not take time to write a paragraph or two on 
their satisfaction or dissatisfaction with a program. Specific and closed-ended questions on topics, 
delivery, and learning would be more appropriate for this type of evaluation. 

Some ERIC components used Web site feedback form to improve service. But some of 
them experienced low response rate also-for instance, one site received a total of 76 user 
comments over a 27-week period. Considering the traffic on ERIC sites, this number is 
problematic. The Web feedback form is a good idea but it will need to be well designed to serve its 
purpose. The site in question provided users with a blank form for input, and that may be why so 
few responses were received. 

In addition to reaching out to users, some components have used Web technology to 
analyze users unobtrusively. Several clearinghouses have begun using Web logs to identify users' 
affiliation (by analyzing the domain name, for example) and track users' pattern of exploration at a 
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site. At least one clearinghouse has analyzed search logs to analyze how end users conduct 
searches. These methods of data collection are relatively easy and fast, and are likely to be used 
more heavily by web sites in the future. 



VI. Discussion 

Data show ERIC information professionals have done much to identify users' information 
needs, made continuous promotion efforts to reach users and non-users, and provided an 
impressive array of services and products. These professionals have employed information 
technology to enhance their services. They appreciate innovations such as AskERIC, web tutorials, 
and NPIN, and aspire to be as helpful as they can be to their users. They demonstrate strong skills 
in identifying, synthesizing, and organizing resources valuable to their constituencies. And they 
show unwavering dedication to serving users. But they are also constrained by limited resources 
and concerned about how to ensure quality services in the face of dwindling support and rising user 
expectations. In the 21st century the ERIC system, given appropriate funding, will include 
educational resources in all formats and offer easy access to anyone involved in the education 
enterprise. User services in this new era will continue to include QA services, identification and 
organization of relevant resources, Web-based services such as listservs and FAQs, publicity 
campaign, and user training and assistance. Many of such services will be enhanced by information 

technology. 

• User Services 

Current user services include an impressive array of services and products (see the list in II. 
Purposes). Many of these services have been identified as critical responsibilities of public services 
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in the digital age (Ferguson & Bunge, 1997; Mardikian & Keseelman, 1995; Sloan, 1998, Sutton, 
1996), and they include many areas where ERIC has the expertise to excel. With a few changes to 
services, ERIC will become more competitive. 

General QA services (Web-based or e-mail based): Many academic and public libraries 
are now providing reference services asynchronously (Davenport, Proctor, & Goldberg, 1997; 

Sloan, 1998), and some have experimented with video conferencing technology to provide 
synchronous services (Davenport, Proctor, & Goldberg, 1997). These libraries learned that video- 
based reference services need to be scheduled to make the best use of staff time, while services 
based on e-mail or the Web have not increased staff work very much (Davenport, Proctor, & 
Goldburg, 1997; Moore, 1998; Sloan, 1998). Many libraries are migrating services to the Web and 
digital reference service has become more prevalent than before. Informal data suggest ERIC user 
services have been successful and AskERIC has offered QA services on the Web. But to remain 
competitive, ERIC will need to seriously consider expanding current user services to the Web. By 
supporting users to submit queries and service requests through the Web, ERIC signals to users 
that it is ready to operate in the digital environment many users have grown used to. A well 
designed Web form can collect information about users and their needs, as the AskERIC form 
illustrates (http://ericir.syr.edu/Qa/userform.html). One of the early adopters of Web-based service 
is the Internet Public Library (IPL), and it has a form that asks the requester to specify the nature of 
the request, how the information will be used, the type of answers preferred, and the types of 
resources consulted (http://www.ipl.org/ref/QUE/RefFormQRC.html). IPL also explains the types 
of requests that are appropriate for Web services and states the turn-around time explicitly to avoid 
user frustration. Many of these services are listed in Sloans paper (1998). 
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ERIC professionals will naturally be concerned about the additional funding such Web- 
based user services entail. Funding, in fact, is a perennial problem for ERIC. It is well known that 
ERIC professionals have been ingenious in stretching their resources to provide services to users 
and many have contributed personal time and labor to making the system work. But for various 
reasons ERIC funding has continued to decrease in spite of increase in federal education 
expenditures. To be viable and competitive, ERIC will need to develop a clear vision of ERIC 
services in the 21 st century and develop a strategic plan to convince funding sources of the 
importance of its services and products and of the need for increase in support. 

Research consultation: Many users have access to technology and can use tools like search 
engines and subject directories to find information, but they soon realize the complexity of the 
information environment is more than they can handle, and many have turned to information 
professionals for assistance. Several libraries have provided this type of service on the Web (see, 
for example, Brown University's Research consultation, 

http://www.brown.edu/FacilitiesAJniversity_Library/forms/ResCnslt.html). ERIC professionals 

have searched commercial databases and Web resources to meet users' needs, so they have the 
expertise and experience in providing research consultation. The key question again is whether 
they are willing to build on their strengths and provide more of such services. The trend in user 
services is to take the services to users. By providing services customized to users' research and 
needs ERIC can ensure its long-term future. 

In addition, there are a few areas ERIC information professionals will need to explore for 
better solutions. A relatively easy task is to provide more coordination in linking related resources, 
such as using the same icons for the same components and placing the icons in the same locations 
across ERIC components. This can be accomplished through an agree-upon template or style sheet. 
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While a cross-site search engine recently provided by AccessERIC can help users explore ERIC 
(see http://www.accesseric.org/resources/search/srchsite.html), it is still necessary to have 
consistent representation of ERIC components and related services (such as the new cross-site 
engine) so that users can take full advantage of ERIC services and products. 

Similarly, the presentation of ERIC interfaces needs to be clarified. Several interfaces have 
been designed to serve different purpose and ERIC components have linked to all or some of these 
interfaces as they see fit. The result is a confusing landscape for users. It would be helpful for 
ERIC components to agree on a way to representing interface options clearly to users to help them 
select the right interface for their questions. The chart at AccessERIC, for example, provides a 
good starting point for this effort. Access to digests is a similar issue. Some components provide 
access only to digests they create, while others also link to the digest archive. A more consistent 
approach would benefit users. 

As indicated above, more visibility for Web-based services and remote consultation 
services, which ERIC has the expertise to provide, will bring more awareness about ERIC and 
generate more interest in and support for the system. But at the 2000 National ERIC Joint 
Directors/Technical Meeting several directors and user services coordinators voiced their concern 
that increase in users will overwhelm the system and affect quality of service. In other words, they 
were concerned about how to provide the same quality service to more users without further 
funding. Again, this is a matter of priority for ERIC. If such services are essential to the new vision 
of ERIC services, then resources must be allocated for them. 

Suppose such services are agreed upon as top priority items, a related issue is how to 
provide such services effectively and efficiently. This issue also received much attention at the 
Joint Meeting. There are at least two models for consideration. The multi-entry model is to provide 
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at individual clearinghouses’ Web sites two forms for request, one for topics in the subject domains 
of the particular clearinghouse (the clearinghouse form), another for any topics related to education 
research (the AskERIC form). In addition, at the ERIC site provided by AccessERIC requests can 
be submitted to a clearinghouse or to AskERIC if the user is not sure which clearinghouse to 
approach for the request. The benefit of this approach is that more service points are provided, and 
users will have more opportunities to submit questions. The drawback is more links to maintain 
and more user requests to answer, which can be a problem if no resources are allocated for such 
services. 

Another model is to centralize all requests to AskERIC, let AskERIC distribute requests to 
clearinghouses according to subject expertise, and delegate to AskERIC the ownership and 
management responsibility of the requests. The benefit is clearinghouses can focus on answering 
questions distributed to them by AskERIC and free themselves from the financial and 
administrative burden of service requests. Users can also benefit from this approach because there 
is only one service point for them to submit inquiries. The drawback is lack of control and heavy 
reliance on one service point for service request. If the service point is incapacitated for any reason, 
the whole system will become paralyzed. The distributed model and the centralized model each has 
its strengths and limitations. This is an issue for the ERIC community to address. 

Many respondents reported strong user interest in full text documents and Internet 
resources. To remain competitive, ERIC database and all products must include Internet resources. 
As one of the directors pointed out, such resources will be more taxing on the system because the 
maintenance of links and archiving of such resources will be costly. Nevertheless, more and more 
resources are available only in electronic format and users are more used to citing Internet 
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resources. As the director of ERIC Facility indicated, this type of resources can be kept in a 
separate file or server to make the management and maintenance of such resources easier. 

Furthermore, ERIC can improve access to Internet resources by using technology to 
provide better information filtering. SourceFinder at the Virginia Military Institute (see 
http://www.vmi.edu/sourcefmder/) and Dickinson College (see 

http://library.dickinson.edu/db_index.html), for instance, have created databases of selected Web 
resources that support users to search by subject, type of materials, keywords, etc. to identify tools 
that are likely to contain the information they need. If ERIC pools Internet resources selected by 
ERIC components into one database, it will give users an important subset of Internet resources to 
use and save them time in searching for educational materials on the Internet themselves. If ERIC 
takes a step further and decides to integrate resources produced by various sources (such as 
teachers, researchers, journalists) into one system for searching, it can follow the example of the 
University of Washington's Digital Library Initiative that uses the CONTENT software suite to 
integrate resources in various formats into one system (http://content.lib.washington.edu). The 
system can be further improve by adding filtering capabilities used by Northern Light 
(http://www.northerlight.com/) that sort search outcomes into subject folders or collection folder to 
help users identify relevant resources. 

It should be noted that the changes recommended here are presented with the assumptions 
named in Section III, Assumptions. That is, it is assumed that ERIC is committed to its role as the 
national center of educational resources and will continue to adopt Web technology to enhance 
services and serve more users. 
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• Evaluation efforts 

Data collected also identify a missing piece in the ERIC system. That is, few evaluation 
efforts have been made and several of them did not produce useful data for planning or service 
improvement. Data suggest the ERIC information professionals may have been too busy providing 
services to establish benchmark for their operation or measure their progress toward ERIC's goals 
and objectives on a regular basis. For instance, ERIC information professionals have employed 
several methods to identify user needs, and used the analysis results to develop products and 
services. These information professionals are confident and competent in doing this task, but there 
has been little effort to assess or document the effectiveness of their service. There is also no data 
for management to determine if a particular service has served its purpose or if more resources are 
needed in a particular area. 

Similarly, ERIC information professionals have made promotion effort in several fronts, 
distributing widely materials about their services and the ERIC system, making numerous 
conference presentations, announcing ERIC services and products on listservs and putting forward 
a strong Web presence. Much hard work has been done to promote ERIC, and many information 
professionals believe they have reached non-users. But there are no clear indicators of the outcome 
of this investment. 

Furthermore, QA services constitute a major part of user services, and ERIC information 
professionals have responded to a large number of requests every year. But even this major service 
has not been analyzed systematically. Use statistics show how many requests are filled each month, 
but the data can not tell us why the current system operates at this level or whether the current level 
is appropriate. Nor can they tell us the quality of the services or users’ opinions on the services. 
Because the staff has been so overextended they have to rely on users' e-mail and thank-you notes 
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to know if good services have been provided. Such feedback is of limited use for planning or 
improvement because, as one respondent pointed out, many users tended to be complimentary and 
offered little for improvement. Recent Web surveys have shed some light on users' overall 
satisfaction with ERIC services and products, but such data still provide inadequate information for 
ERIC information professionals to improve or plan their services. 

Training workshops were also not evaluated, and information professionals have had no 
diagnostic tool to assess their preparation and delivery or to test the audience's understanding of the 
subjects covered in workshops. 

All ERIC components have attempted some form of formal or informal evaluation with 
varying success. The data collected indicate who ERIC users are, what they use the system for, and 
if they are satisfied with ERIC services and products, but no data have been collected to help 
individual ERIC components assess their output, the outcome of their services, and the value of 
their services. A number of factors may have contributed to this state: 

1) limited resources may have kept ERIC components from assessing their services in-depth; 

2) uncertainty over the benefits of evaluation may have made ERIC components unwilling to 
conduct evaluation; 

3) lack of experience in evaluation research may have delayed more systematic evaluation; and 

4) uncertainty over management's commitment to evaluation may have provided little incentive 
for ERIC components to undertake costly evaluation. 

For serious evaluation to take place, these issues will need to be addressed first. 

All ERIC components have worked extremely hard and produced interesting services and 
products. And yet, there is no concrete data that can inform decision makers what services to 
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enhance, remove, or add to the system. Nor is there any data on the impact of services. But ERIC 
components have kept some statistics, some of which, such as user type, if coupled with use 
pattern, could produce more useful information about how different types of users access ERIC or 
use ERIC resources. Some of the earlier evaluation instruments could also be improved or 
extended to generate more useful data — the evaluation form at the Urban Education site, for 
example, has attempted to identify services most important to users. So evaluation of ERIC user 
services is not an impossible task. What is critical is the system must be very clear from the outset 
what it intends to accomplish through an evaluation. As ERIC strives to become the premier 
education information center and to meet the needs of many users such data will become even 
more important not only for ERIC managers to plan and allocate resources, but also to justify 
increase in funding and the introduction of new program and services. 



VII. Evaluation of ERIC user services 

• Benefits of evaluation 

Successful evaluation of services requires the involvement of all ERIC components because ERIC 
information professionals know their services and products and can contribute much to planning 
the evaluation and collecting data. If management is committed to action after evaluation data 
become available, evaluation can benefit ERIC in the following ways, as suggested by evaluation 
literature (Griffiths & King, 1991; Hemon & McClure, 1990; Lancaster, 1993; Van House et al., 
1987; Van House, Weil & McClure, 1990): 

o Understand current level of service 

o Identify problematic areas 
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o Monitor progress toward service objectives 
o Justify internal allocation of resources 

o Demonstrate commitment to excellence in service to external parties 
o Demonstrate accountability to external parties 

• Evaluation cycle 

User services evaluation should be designed to assist in the planning and management of 
services. Figure One illustrates that evaluation is a cyclical process. One begins by identifying a 
target area for evaluation, specifies a desired level of performance, select measures to assess 
performance level, collects data, analyzes data, and uses the findings to make necessary changes or 
to identify a new target area for investigation (Bunge, 1994; Robbins-Carter & Zweizig, 1985; Van 
House et al., 1987; Zweizig, 1984). In conducting research one needs to pay close attention to the 
issues of validity, reliability, and generalizability, and these issues are of particular importance 
when one solicits responses from users. Validity is concerned with whether a variable actually 
measures what it says it measures. For instance, "hit counts" is a valid measure for how many times 
a web page is visited, but not a valid measure for the importance of a Web page because the 
number can be manipulated through web page design. To ensure validity the evaluators will need 
to define variables clearly and operationalize them. Reliability is concerned with whether a variable 
will generate the same response over time. For instance, instead of asking people how old they are, 
the evaluator could ask them to write down their birth years to ensure reliability. Generalizability is 
concerned with whether a finding can be generalized to a larger population, and a standard method 
to ensure generalizability is to take a random sample of a population. If data are not collected from 
a random sample, it will be risky to generalize findings to the broader user population. The low 
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response rates of several of the surveys conducted in the ERIC system therefore render the findings 
not useful. 

For the purpose of identifying an area for evaluation, the method tested by Broadbent and 
Lofgren holds much promise. The researchers have successfully used the Priority and Performance 
Evaluation (PPE) approach to solicit input from users, managers, and information service staff on 
services, identify priority areas for service, assess current performance, and identify areas for 
improvement (1993; Lofgren, 1992). As Figure Two shows, in PPE one asks each subject to 
prioritize a service and to assess the performance of that service. The data are then charted to 
determine the status of the service. A service can fall in the SUCCESS quadrant (high priority, 
high performance), IMPROVEMENT quadrant (high priority, low performance), ACCEPT ABLES 
quadrant (low priority, low performance), or TROUBLES quadrant (low priority, high 
performance). Services belonging to the IMPROVEMENT quadrant should be investigated to 
determine causes for low performance, and services in the TROUBLES quadrant warrant a review 
to determine if they should be dropped. 

• Factors affecting user services 

Many factors affected user services, and a decentralized environment like ERIC is even more 
challenging because there are at least 16 to 20 sets of variations involved in user services. 
Literature on public services has identified the many factors that may affect user services (e.g., 
Hemon & McClure, 1990; Lancaster, 1993; Murfin & Gugelchuck, 1987; White & Abels, 1995). 

• Service policy : Policies on the cost of service, the time allowed to perform a service, and 
service priority can affect service quality. 
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• Collection/resources: The size of a collection, how that collection is organized, and how 
resources are accessed also contribute to service quality. 

• Staff: Staffs knowledge, training, experience, communication skills, and other related 
qualities play important role in service. 

• Service inquiries: The subject of a service request and its complexity also affect service 
result. 

• Users: Users’ attitude toward information professionals and information seeking, their 
expectations, and their ability to communicate their needs are important factors. 

• Environment: The physical facility in which services are provided, the equipment, and staff 
morale are also important factors in service. 

What these factors illustrate is the need to put evaluation data in context. It would be unwise to 
judge performance by looking at absolute numbers or one single measure (such as satisfied or 
unsatisfied). The context in which a service is provided will help shed light on how and why a 
certain level of service is provided. 

• Strategies for evaluating in a decentralized environment 

Several strategies can be used to ensure good evaluation in a decentralized environment like 
ERIC. The most important task is to specify the goals and objectives of the evaluation as clearly as 
possible. They will in turn guide the design and execution of an evaluation, prepare participating 
staff and agencies for the evaluation, and keep the investigation on track. Specifically the following 

tasks should be done: 
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1) Decide from the outset whether the evaluation will be done at the system level or the 
component level, because this decision will affect sample size and how the sample is 
taken. 

2) Measures must be carefully selected, defined and operationalized in unambiguous terms 
to avoid confusion. 

3) Data sources will need to be clearly identified. 

4) Data collection procedures must be spelled out clearly. 

5) Data collection personnel will need to be trained. 

6) An evaluation coordinator, or a small team, should be responsible for resolving 
questions about data collection. 

7) The evaluation coordinator/team should keep logs of data collection, including the 
progress, problems, and solutions. 

With these steps in place, one has a much better chance for success in evaluating a distributed 
system such as ERIC. 

• Evaluation literature 

More than 30 years of literature on public services evaluation can be drawn on to identify 
promising measures and data collection methods. Whitlatch provides an overview of methods for 
evaluating reference service and discusses strategies for data collection and analysis (1992). Murfin 
summarizes the history of user evaluation of reference service, advocates the inclusion of users in 
the process, and recommends that multiple methods be used to evaluate reference services 
appropriately (1995). Aspects of user services frequently evaluated include 
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• Accuracy of answers (e.g., Bunge, 1990; Crews, 1988; Elzy et al., 1991; Hernon & 
McClure, 1987): studies of this category examine the accuracy of answers, using methods 
such as simulation, unobtrusive observation, and obtrusive testing. The unobtrusive method 
has received much attention because it uncovered that the reference staff could answer 
about 55% of the factual questions posed to them. Limitations of this approach have been 
noted by many researchers (e.g., Bailey, 1987; Crews, 1988; Tyckoson, 1992), but the 
approach is believed to be effective in identifying knowledge gaps in staff (Baker & 
Lancaster, 1991). 

• Librarian behavior and communication skills (e.g., Bicknell, 1994; Cullen, 1992; Gers 
& Seward, 1985; Humphries & Naisawald, 1991; Jardine, 1995; Kleiner, 1991; Radford, 
1996; Ricks et. al., 1 99 1 ; Seay, Seaman, & Cohen, 1996; Whitlatch, 1990). Studies of this 
type focus on librarian behavior in providing services, and some have examined whether 
librarians’ behavior is related to how users evaluate a service. User survey, librarian survey, 
and simulation have been used to collect data. 

• Environment (e.g., Bicknell, 1994; Cullen, 1992; Humphries & Naisawald, 1991; 
Mendelsohn, 1997; Seay, Seaman, & Cohen, 1996). Literature in this area covers the 
collection size, the organization of a collection, physical facilities, equipment, and other 
environmental factors to see how they may be related to service quality or users perception 
of service quality. User survey and librarian survey have been used in this type of research. 

• User satisfaction (e.g., Andaleeb & Simmonds, 1998; Bicknell, 1994; Cullen, 1992; 
D’Elia & Walsh, 1983; Griffiths & King, 1991; Murfin & Gugelchuck, 1987; Murfin, 1995; 
Whitlatch, 1 990). Literature in this area affirms the value of user input in the evaluation of 
user services. Studies examine factors that may have contributed to user satisfaction, 
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present examples for soliciting valid report of user satisfaction, point out the need to 
include librarians’ views in the evaluation, and demonstrate how meaningful data can be 
derived by combining user satisfaction with other measures such as use frequency and turn- 
around time. User survey, librarian survey, and critical incident reports have been used for 
these purposes. This variable is likely to be of most value to ERIC decision-makers. 

• Value and impact of services (e.g., Abels, 1997; Broadbent & Lofgren, 1993; Griffiths 
& King, 1991; Marshall, 1992, 1993) 

These studies use surveys and critical incidents to solicit users’ view on services and ask 
them to estimate the dollar value of services or to state the impact of services. Griffith & 
King show that measures can be combined to calculate an estimate for the cost of offering a 
service, the cost of offering an alternative service, and the cost of no service at all (1991). 
Broadbent & Lofgren analyze the cost-effectiveness of a current awareness service (1993). 
Marshall’s subjects reported the impact of medical information on physicians (1992) and 
the impact of financial information on decision makers’ work in Canadian banks (1993). 



• Evaluation types and levels 

King & Bryant (1971) describe two important types of evaluation: macroevaluation and 
microevaluation. Macroevaluation measures how well a system is functioning, such as how 
satisfied users are with ERIC system. But it does not explain why users are satisfied or why they 
are not. Microevaluation measures how a system is functioning and why it is functioning at that 
level. Depending on the purposes of evaluation, sometimes macroevaluation may be sufficient. But 
most evaluation studies include both macroevaluative measures and microevaluative measures to 
gain a clearer picture of a system's operation. 
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Information services can be evaluated at three levels: effectiveness, cost-effectiveness, and 
cost-benefits (Baker & Lancaster, 1991). Effectiveness refers to how well a service meets the need 
of a user. When a user requests a full text from EDRS online and receives the document, that is an 
indication of effectiveness. Cost-effectiveness refers to the efficiency of a service in meeting a user 
need. So if the same request is filled and it costs EDRS $10 to provide that service, a cost-effective 
measure can be calculated to determine the system's internal operating' efficiency. Cost-benefits 
refer to the value (benefits) of a service and whether the cost of providing that service is justified 
by the benefits. Because benefits may not be immediate or tangible, this type of study is more 
difficult to conduct than the other two. Nevertheless, it is possible to ask users to articulate the 
perceived benefits of a service. 

• Evaluation of ERIC user services: Three approaches 

ERIC user services can be evaluated in a number of ways. Figure Three presents three options. 
OUTPUT ANALYSIS 

The first type, output analysis, is the easiest one to conduct because it is system-centered 
and much of the data can be collected internally. User services are assessed in terms of how well 
they meet the needs of users ("effectiveness") and how much it costs the system to operate at this 
level of effectiveness ("cost-effectiveness"). Macroevaluative measures show what services are 
offered and how satisfied users are with them. The focus is on system output, i.e., what services 
and products are produced, and the data is descriptive and quantitative. For instance, how many 
questions were asked during a given period? What is the cost of answering these many questions? 
How many reference requests were filled successfully during this period? What is the cost of 
successful reference services? Output information is usually collected by regular record-keeping, 
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which can be done by machines (search logs and web logs) or by humans (spreadsheet or record 
books). To support comparative analysis later it is critical that data categories be clearly defined 
and operationalized. For instance, "training" can be defined as "teaching users how to make use of 
ERIC services and products, especially the ERIC database." To operationalize this variable the 
evaluators will need to specify how training activities will be counted for instance, is training by 
tutorial counted the same as one-on-one instruction? 

Because many ERIC component have kept statistics on their various service activities, this 
type of evaluation will be an extension of what they did before. That is, cost data can be combined 
with output statistics to determine the cost of each type of service. In addition, quality assessment 
data (such as filled reference requests) and user satisfaction measures can be combined with output 
statistics to determine the level of effectiveness; and the same set of data can be combined with 
cost data to calculate the cost for offering successful services and producing satisfied users. It will 
be critical in a decentralized environment like ERIC to standardize data categories and reporting 
procedure across ERIC components. 

USAGE ANALYSIS 

The second type of evaluation is more difficult than the last one because it is user-centered 
and most of the data are collected from users. User services are assessed for how they meet the 
needs of users, and the cost of meeting such needs ("cost-effectiveness"). To understand how the 
services meet user needs, this type of study will examine the context in which users seek 
information, will review users' experience in using the services, and will identify the immediate 
outcome of their information use. Some of the data will be descriptive (such as 20 postsecondary 
student users), while other data will be diagnostic (such as only 25% of the information provided is 
relevant). This type of study will generate both quantitative and qualitative data. 

O 

ERIC 
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Typically data can be collected through surveys and structured interviews- structured ones 
are more focused and easier to analyze. Focus group interviews are used frequently in marketing 
research and product testing, but not as often in product evaluation. Nevertheless, this method is 
good for identifying topic area for investigation and for generating questions for survey 
instruments. 

In addition to these methods, data on whether user needs are met can be collected from less 
obtrusive methods. For instance, the quality of a service can be evaluated by an objective evaluator. 
A supervisor can assess a search result for accuracy, completeness, and currency, but subject 
measure such as "relevancy" may require input from the user. Another unobtrusive method to 
assess user services such as QA services is to pose requests through proxies and use the result to 
evaluate the quality of a service and the knowledge of the service provider. This method is useful 
for identifying a staff member's gap of knowledge in a particular area. It is most effective if 
combined with further training in an identified problem area. Hernon & McClure (1987) have 
explained the procedures for this type of analysis and addressed the validity and reliability issues 

in-depth. 

Content analysis is an effective way to identify user needs and the data can be verified with 
data collected from other methods such as focus group interviews or surveys. Web logs and search 
logs are fairly new and can provide new insights on navigation patterns and resource usage 
patterns. But such data should be interpreted carefully. For instance, if a Web site is designed in 
such a way that a visitor needs to return to the home page to get to other categories of information, 
the "hits" for the home page may become artificially high. The evaluators will need to take that into 
account in data analysis. Similarly, logs can tell us how many times a resource link is visited or 
how many times a search operator is used, but they can not tell us "why" the activity takes place. 
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Several sites have provided an e-mail or a feedback box area for users to send in comments, 
questions, and suggestions. This type of data can be useful but the number of inputs may not be 
high. 

Some of the data categories collected for this type of study may include: Who the users are 
(teachers, parents, etc.), what their needs are (parenting tips or lesson plans), how they plan to use 
the information (presentation or course preparation), how the services are provided (speed, 
courtesy, etc.), how they access the services (phone or Web), what services are provided 
(bibliography, Digest, etc.), quality of service (relevancy of information, turn-around time, etc.), 
satisfaction with the service, and satisfaction with how the service was provided. 

IMPACT ANALYSIS 

The third type of evaluation is most challenging and costly to conduct because the value of 
a service can be difficult to assess. Nevertheless, this type of study, if done well, can provide 
tremendous support for justifying funding for services. Impact analysis is user-centered. 
Macroevaluative measure and microevaluative measures are both collected to understand the long- 
term outcomes of a particular service. Data can be descriptive, indicating how the service affects 
the user, and diagnostic, indicating how a service fails or succeeds. Both qualitative data and 
quantitative data are collected. 

Interviewing is an effective but expensive method to collect impact data because it allows 
the user to describe the entire information seeking process and put an information service in that 
context. Typically unstructured interviews are used to let the user describe freely and voluntarily 
how the information received has affected him or her. Case studies using the critical incident 
approach is one of the more popular methods to understand the effects of a service. The study 
usually begins with the user recalling a recent problem or concern related to their work. The user is 
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then encouraged to describe the important aspects of the problem, and identify things they wanted 
to know in order to solve the problem. For each thing identified, the user is asked to describe how 
he or she found out about it, whether the found information was helpful, if it did help, how was it 
helpful, and if it did not, why it was not helpful. Content analysis is performed on collected data to 
gain a picture of how users solved problems and how a service helped him. The impact aspect is 
usually assessed at the personal level and the professional level. Some of the impact measures at 
the personal level include knowledge gained from the information service, time saved, money 
saved, more confidence in problem solving, and informed decision. Impact measures at the 
professional level usually include recognition at work, reward (sometimes financial), increased 
mobility in job, and promotion. 

Surveys are a less expensive alternative for assessing impact of user services. Griffith & 
King offer numerous ways to collect data from users and the system to estimate the cost of a 
service, the cost of an alternate service, and the cost of no service (1991). Marshall’s survey of 
physicians revealed impact of medical information on them (1992). A similar study in financial 
institutions asked decision-makers to report the impact of financial information on their work 
(1993). A recent AskERIC's survey asked users to indicate whether the service save them time 
(1998). These examples show how various aspects of impact can be measured. If ERIC undertakes 
a study of the impact of its services and products, it will be necessary for the evaluation team to 
specify the types of impact it intends to measure first. Then a data collection method can be 
identified. 
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VIII. Conclusion 

Data on ERIC user services show that ERIC information professionals have done much to 
identify users’ information needs, made continuous promotion efforts to reach users and non-users, 
and provided many services that are important in the digital age. Several changes are recommended 
in this paper. By applying new information technologies to current services ERIC will enhance 
those services. By expanding its coverage to Internet resources, full texts, and maybe practitioners 
and field experts, ERIC will demonstrate its commitment to meeting the needs of users and become 
a more competitive player in the information arena. These efforts will help ERIC establish itself as 
The national information center of educational resources and enable it to have greater impact on 
education and educational research. 

But before the recommendations can be implemented, the funding issue must be addressed. 
Armed with a clear vision of ERIC services in the new century and data on the values and impact 
of ERIC services and product, ERIC managers will be able to present a strong case for increasing 
Federal funding for the ERIC system. Drawing on the data on ERIC user services and the literature 
on public services evaluation, the investigator describes a series of steps to ensure success in 
evaluating ERIC user services and offers three evaluation options for the ERIC community to 
consider. Although the ERIC system is complex and offers a wide range of services and products, 
an evaluation of user services in this decentralized environment does not require large sets of study 
instruments. Because ERIC components provide similar types of user services, it is possible for 
them to start with the same instrument. For instance, a survey template with commonly agreed 
upon data categories can be created for individual ERIC components to add questions unique to 
their settings. Then the findings can be used to compare performance across components and guide 
individual clearinghouses in improving services. ERIC information professionals have done 
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tremendous work in providing user services, but their efforts have not been assessed 
systematically. If proper evaluation is conducted, Central ERIC and ERIC components will have 
benchmarks for the entire operation and will have a solid basis from which to determine the 
allocation of resources and to plan for the future of the ERIC system. 
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Evaluation Cycle 



Target Area 




Figure 1. Evaluation Cycle 




55 



51 



Priority and Performance Evaluation Method 

Sample Questions 

In your opinion, what priority should ERIC assign to each of the items below? 

Please circle the number that best describes your assessment. 

Low Priority High Priority 

Timely Service 1 2 3 4 5 6 7 

Understanding user needs 1 2 3 4 5 6 7 

In your opinion, how well have ERIC performed in each of these areas? 

Please circle the number that best describes your assessment. 

Poor 

Timely Service 12 3 4 5 

Understanding user needs 12 3 4 5 



Excellent 
6 7 

6 7 



PPE Chart 
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Figure 2. PPE Method 
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Output Analysis 


Usage Analysis 


Impact Analysis 


Level 


Effectiveness & Cost- 
Effectiveness 


Cost-Effectiveness 


Cost-Benefits 


Type 


Macroevaluation 
(WHAT, HOW 
WELL) 


Microevaluation 
(WHO, HOW, WHY) 


Macroevaluation & 

Microevaluation 

(HOW) 


Perspective 


System 


Users 


Users 


Focus 


Output 


Context & experience 
of use; Immediate 
outcome 


Long-term outcome 
(HOW) 


Nature 


Descriptive 


Descriptive & 
Diagnostic 


Descriptive & 
Diagnostic 


Data type 


Quantitative 


Quantitative, 

Qualitative 


Qualitative, 

Quantitative 


Data collection 


1 . Statistics 

2. Search logs 

3. Web logs 


1 . Surveys 

2. Structured interviews 

3. Focus group 
interviews 

4. Quality assessment 

5. Content analysis 

6. Web logs 

7. Search logs 

8. Feedback box 


1 . Surveys (limited) 

2. Interviews 

3. Case studies (critical 
incidents) 


Data categories 
(examples) 


1. Services provided 

2. Services used 

3. Use frequency by 
service 

4. Number of users 

5. Level of operation 
(overall) 

6. Cost by service 
type 


1 . Information needs 
(of users and non-users) 

2. User background 
(position, level, etc.) 

3. Experience of use 
(process and outcome: 
Purposes, product, 
quality of services, 
satisfaction) 


1 . Impact at personal 
level (knowledge 
gained, time saved, 
money saved, better 
decision) 

2. Impact at 
professional level 
(recognition, reward) 



Figure. 3. ERIC User Services Evaluation Options 
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Appendix 1. Questions for ERIC Directors 



***** Questions on ERIC User Services ***** 

A. USERS 

A1 . USERS: Do you have information on your users in terms of their types, purposes of using 
your services, frequency of usage, access methods (e-mail, phone, and/or Web), satisfaction with 
your services, and measurable impact of your services? 

A2. NEW USER TYPES: Have you tried to reach out to non-traditional users such as users with 
disability and parents of home-schooled children? If so, how have you reached out to them? What 
lessons can you draw from such experience? 

A3. NON-USERS: Do you know of users who could benefit from ERIC services but have not 
used the ERIC system? If so, how do you learn of them? Do you know why these users have not 
tried ERIC? What do you believe are the best ways to reach non-users? 

A4. USER NEEDS: How do you learn about the needs of your users? 

A5. UNMET NEEDS: Based on your experience, what needs of users remain to be met by your 
center? 



B. QUESTION/ANSWERING SERVICES 

Bl. QA Services: How have you provided Q/A services? Please include all the methods you have 
used. If you have evaluated these methods, please share your evaluation results with me. 

B2. Comparison of QA services methods: Do you agree that QA services by e-mail, by phone, in 
person, or by the Web work equally well? If not, what types of questions can be best addressed by 
e-mail? What types of questions are most suitable for Virtual Reference Desk? 

B3. AskERIC vs. individual clearinghouse services: Have you provided your clearinghouse user 
services by e-mail? In your assessment, how helpful is AskERIC for your users? Can AskERIC 
replace individual clearinghouse user services or referral to services and resources in local 
libraries? 

B4. How do you feel about providing user services through Virtual Reference Desk? Do you plan 
to establish VRD in the near future? Do you have any concerns? If so, what are they? 

B5. Who among your staff provides QA services? How are they prepared? Do you have standards 
or guidelines for providing QA services? How do you ensure the quality of the services provided 
at your center? 
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B6. Have you evaluated your QA services? If so, please share your evaluation results with me. In 
addition, please indicate how satisfied you are with the evaluation methods. Are there things you 
would like to change in your next evaluation? 



C. SEARCH SERVICES 

Cl . User training: If you train users to search ERIC, how do you do that? Do you use a tutorial? 

Is it a one-on-one instruction session? Do you assess how well the users learn to search ERIC? 

C2. Do you log users' online searches and analyze them for search patterns and hot topics? Do you 
keep a log of activities at your Web site and analyze them for user behavior and needs? If so, 
please share your findings with me. 

C3. How do you feel about the current search engine and search interface for ERIC? Are there 
changes you would like to make? 

C4. How do you feel about the current ways ERIC delivers search results to users? Are there 
changes you would like to make? 



D. WEB: POTENTIAL & CHALLENGES 

D1 . Web technology offers a lot of possibilities for new types of user services. Please describe up 
to three new types of user services you have provided through the Web. In addition, please 
describe two new user services you would love to provide through the Web if you have the 
resources. 

D2. Internet has brought users around the globe to ERIC. What are the challenges in serving 
international users? How have you addressed some of these challenges? What are the areas you 
would like the ERIC system to address? 

D3. Some have expressed a wish to offer ERIC services anytime anywhere. Do you share that 
vision? What will it take to realize this vision? 

D4. Some believe that ERIC staff should go beyond just providing answers and information to 
users. They feel that teaching users to search for information and evaluate information will serve 
users better in the long run. How do you feel about this view? 

Will the Web make it easier for us to teach users? 

D5. Web technology raises user expectations of ERIC. Please describe what you have observed 
about user expectations. 
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Appendix 2. Questions for ERIC User Services Coordinators 



***** Questions for User Services Coordinators ***** 

1 . What user services are provided at your clearinghouses now? 

2. How are services provided? What methods do you use to provide services? What guidelines or 
mechanisms do you use to ensure quality of your services? 

3. Who are providing your user services? How are they prepared? How are they supported for 
their work? 

4. How have user services been evaluated so far? How do you feel about the evaluation completed 
so far? What changes, if any, would you like to make in your evaluation? Have you measured the 
impact of your services? (If you have any evaluation instrument, please send me a copy.) 

5. What do you need to provide better services? What issues do you believe ERIC should address 
to provide quality user services in the next century? 
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